This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle and only multimodal models can succeed: difficult examples ("benign confounders") are added to the dataset to make it hard to rely on unimodal signals. The task requires subtle reasoning, yet is straightforward to evaluate as a binary classification problem. We provide baseline performance numbers for unimodal models, as well as for multimodal models with various degrees of sophistication. We find that state-of-the-art methods perform poorly compared to humans, illustrating the difficulty of the task and highlighting the challenge that this important problem poses to the community.

Different people may have different opinions for the same memes. Must the opposite of the hateful speech be harmless ones? Hate speech detection is NOT a strictly binary classification problem. Hence, the modeling for the task in the paper is inaccurate. Could the authors present a candidate solution for it?

This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle and only multimodal models can succeed: difficult examples ("benign confounders") are added to the dataset to make it hard to rely on unimodal signals. The task requires subtle reasoning, yet is straightforward to evaluate as a binary classification problem. We provide baseline performance numbers for unimodal models, as well as for multimodal models with various degrees of sophistication. We find that state-of-the-art methods perform poorly compared to humans, illustrating the difficulty of the task and highlighting the challenge that this important problem poses to the community.

Memes on the Internet are often harmless and sometimes amusing. However, by using certain types of images, text, or combinations of both, the seemingly harmless meme becomes a multimodal type of hate speech -- a hateful meme. The Hateful Memes Challenge is a first-of-its-kind competition which focuses on detecting hate speech in multimodal memes and it proposes a new data set containing 10,000+ new examples of multimodal content. We utilize VisualBERT -- which meant to be the BERT of vision and language -- that was trained multimodally on images and captions and apply Ensemble Learning. Our approach achieves 0.811 AUROC with an accuracy of 0.765 on the challenge test set and placed third out of 3,173 participants in the Hateful Memes Challenge.

Looking at the history of mankind spanning thousands of years, hatred was always present and people have been persecuted for a wide variety of reasons up until this day, but in recent times hate has been digitized and weaponized with the rise of social media. One of the biggest roadblocks law enforcement agencies and private institutions face when attempting to detect then combat hateful material found on social networks, especially Twitter, is how exactly to deal with the massive amount of data, including tons of false positives, sarcastic posts, emerging trends that go viral within hours as well as many other variables. It's a monumental task that no analyst, army of analysts or social media intelligence tools of the past can accomplish. For the past 2 years, the team at Soteria Intelligence has focused on developing technologies to confront online hate, school threats, terrorist propaganda and other challenges we face in a very unorthodox way, and through our research and development it became clear that the only way to solve the complex problem at hand was to use deep learning and machine learning. Taking 10 years of research on social media behavior and 5 years of research on social media threats in particular, along with input from a wide range of subject-matter experts, we've focused on creating machine learning systems with ability to assess social media activity faster and more accurately than humanly possible.